Add: Support Custom Sequence as a Backend of an API #12524

BLasan · 2024-08-25T17:50:56Z

Purpose

Some API developers want to use connectors within the api manager to implement mediation logic as they do not want to use an additional MI deployment if their use cases are simple. Within the API Manager, currently, we do not have a construct that will allow us to facilitate this requirement.

Approach

Provide a new Endpoint Type "Sequence Backend" for REST APIs under Endpoint Tab
Allow Sequences to be added for Sandbox and Production Key types separately,.
Perform download, delete, upload operations for Sequence Backends.
Add support for APICTL import and export scenarios for REST APIs
Introduced a new table to store Sequence Backend. Following is the schema for the table.

CREATE TABLE IF NOT EXISTS AM_API_SEQUENCE_BACKEND (
  ID VARCHAR(60) NOT NULL,
  API_UUID VARCHAR(256) NOT NULL,
  REVISION_UUID VARCHAR(256) DEFAULT '0',
  SEQUENCE LONGBLOB NOT NULL,
  NAME VARCHAR(256) NOT NULL,
  TYPE VARCHAR(120) NOT NULL,
  PRIMARY KEY (ID),
  FOREIGN KEY (API_UUID) REFERENCES AM_API(API_UUID) ON DELETE CASCADE
)ENGINE INNODB;

Affected Flows

Please refer - wso2/api-manager#3007 (comment)

Issue

Add Custom Backend for APIs api-manager#3007

CLAassistant · 2024-08-25T17:51:02Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

AnuGayan · 2024-09-12T11:35:09Z

...ts/apimgt/org.wso2.carbon.apimgt.rest.api.publisher.v1/src/main/resources/publisher-api.yaml

@@ -487,6 +487,218 @@ paths:
          -H "Content-Type: application/json" -d @data.json
          "https://127.0.0.1:9443/api/am/publisher/v4/apis/7a2298c4-c905-403f-8fac-38c73301631f/topics"'

+  /apis/{apiId}/custom-backend/{customBackendId}/content:


rename to sequence-backend/{sandbox/Prod}/{sequenceID} check dev portal key type implementation

AnuGayan · 2024-09-12T11:36:41Z

...ts/apimgt/org.wso2.carbon.apimgt.rest.api.publisher.v1/src/main/resources/publisher-api.yaml

+              schema:
+                type: string
+          content:
+            application/zip:


Change the content type

AnuGayan · 2024-09-12T11:37:51Z

...ts/apimgt/org.wso2.carbon.apimgt.rest.api.publisher.v1/src/main/resources/publisher-api.yaml

@@ -487,6 +487,218 @@ paths:
          -H "Content-Type: application/json" -d @data.json
          "https://127.0.0.1:9443/api/am/publisher/v4/apis/7a2298c4-c905-403f-8fac-38c73301631f/topics"'

+  /apis/{apiId}/custom-backend/{customBackendId}/content:


lets remove the sequenceID

AnuGayan · 2024-09-12T11:41:51Z

...ts/apimgt/org.wso2.carbon.apimgt.rest.api.publisher.v1/src/main/resources/publisher-api.yaml

+              schema:
+                type: string
+          content:
+            application/json:


Make this a text type

AnuGayan · 2024-09-12T11:55:59Z

...mgt/org.wso2.carbon.apimgt.impl/src/main/java/org/wso2/carbon/apimgt/impl/dao/ApiMgtDAO.java

+                endpointConfig.put("sequence_id", resultSet.getString("ID"));
+            }
+        } catch (SQLException ex) {
+            handleException("Error when retrieving Custom Backend of API: " + apiUUID, ex);


Return the relevant exception codes

AnuGayan · 2024-09-12T11:56:37Z

...mgt/org.wso2.carbon.apimgt.impl/src/main/java/org/wso2/carbon/apimgt/impl/dao/ApiMgtDAO.java

+            resultSet = ps.executeQuery();
+            while (resultSet.next()) {
+                map.put("type", resultSet.getString("TYPE"));
+                map.put("sequence_name", resultSet.getString("NAME"));


define and use Static variables

BLasan · 2024-09-19T01:11:11Z

Only websub tests are failing now

BLasan · 2024-09-21T02:04:22Z

BLasan · 2024-09-23T02:15:33Z

Changes were not from this PR

AnuGayan · 2024-09-23T03:49:41Z

.../apimgt/org.wso2.carbon.apimgt.api/src/main/java/org/wso2/carbon/apimgt/api/APIProvider.java

@@ -65,6 +66,10 @@ public interface APIProvider extends APIManager {
    Comment getComment(ApiTypeWrapper apiTypeWrapper, String commentId, Integer replyLimit, Integer replyOffset) throws
            APIManagementException;

+    void deleteCustomBackendByID(String apiUUID, String type) throws APIManagementException;


Lets add Java doc comments. Check other places as well

AnuGayan · 2024-09-23T03:51:14Z

...o2.carbon.apimgt.api/src/main/java/org/wso2/carbon/apimgt/api/model/SequenceBackendData.java

@@ -0,0 +1,58 @@
+package org.wso2.carbon.apimgt.api.model;


Add license header

AnuGayan · 2024-09-23T03:55:41Z

.../src/main/java/org/wso2/carbon/apimgt/gateway/handlers/common/APIMgtLatencyStatsHandler.java

-                        .BACKEND_REQUEST_START_TIME));
-                messageContext.setProperty(APIMgtGatewayConstants.BACKEND_LATENCY, System.currentTimeMillis() -
-                        executionStartTime);
+            if (APIUtil.isAnalyticsEnabled()


How will this be applied in Sequence as a backend scenario

AnuGayan · 2024-09-23T03:56:16Z

...t/org.wso2.carbon.apimgt.impl/src/main/java/org/wso2/carbon/apimgt/impl/APIProviderImpl.java

@@ -21,6 +21,7 @@
 import com.fasterxml.jackson.core.JsonProcessingException;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.google.gson.Gson;
+


remove this new line

AnuGayan · 2024-09-23T03:59:24Z

...mgt/org.wso2.carbon.apimgt.impl/src/main/java/org/wso2/carbon/apimgt/impl/dao/ApiMgtDAO.java

+                    addPstmt.setString(6, rs.getString("NAME"));
+                    addPstmt.addBatch();
+                    count++;
+


Why these are commented out

AnuGayan · 2024-09-23T04:06:50Z

...mgt/org.wso2.carbon.apimgt.impl/src/main/java/org/wso2/carbon/apimgt/impl/utils/APIUtil.java

+     * @return The Sequence of the Custom Backend as an Input Stream
+     * @throws APIManagementException If an error occurs while reading, throws an error
+     */
+    public static InputStream getCustomBackendSequence(String extractedFolderPath, String customBackendFileName,


Why are we returning an InputStream? where are we closing this?

AnuGayan · 2024-09-23T04:09:11Z

...n/src/main/java/org/wso2/carbon/apimgt/rest/api/publisher/v1/common/TemplateBuilderUtil.java

+                    }
+                }
+            } catch (IOException | ParseException ex) {
+                throw new APIManagementException("Error when updating Endpoint Configuration", ex);


Lets add more details to this error, API name etc.

BLasan · 2024-09-23T07:59:38Z

BLasan · 2024-09-26T01:57:11Z

Build is passing

RakhithaRR · 2024-09-26T05:00:39Z

...mgt/org.wso2.carbon.apimgt.impl/src/main/java/org/wso2/carbon/apimgt/impl/dao/ApiMgtDAO.java

+            addCustomBackend(apiUUID, sequenceName, null, sequence, type, connection, backendUUID);
+            connection.commit();
+        } catch (SQLException e) {
+            handleException("Error while adding Custom Backend for API : " + apiUUID, e);


We might need to add a rollback in case of an exception. Check other places as well.

RakhithaRR · 2024-09-26T05:06:35Z

...mgt/org.wso2.carbon.apimgt.impl/src/main/java/org/wso2/carbon/apimgt/impl/utils/APIUtil.java

+            String fileName = extractedFolderPath + File.separator + sequenceName;
+            if (checkFileExistence(fileName)) {
+                if (log.isDebugEnabled()) {
+                    log.debug("Found policy definition file " + fileName);


This should be not policy but custom backend right?

RakhithaRR · 2024-09-26T05:09:11Z

features/apimgt/org.wso2.carbon.apimgt.core.feature/src/main/resources/sql/postgresql.sql

@@ -1858,6 +1858,17 @@ CREATE TABLE IF NOT EXISTS AM_API (
    UNIQUE (API_UUID)
 );

+CREATE TABLE IF NOT EXISTS AM_API_SEQUENCE_BACKEND (


We need to add the database changes to multi-dc scripts as well.

AnuGayan · 2024-09-26T05:37:22Z

...common/src/gen/java/org/wso2/carbon/apimgt/rest/api/publisher/v1/dto/SequenceBackendDTO.java

@@ -0,0 +1,142 @@
+package org.wso2.carbon.apimgt.rest.api.publisher.v1.dto;


Lets add the license header. Check other places as well

AnuGayan · 2024-09-26T05:37:52Z

...common/src/gen/java/org/wso2/carbon/apimgt/rest/api/publisher/v1/dto/SequenceBackendDTO.java

+import io.swagger.annotations.*;
+import java.util.Objects;
+
+import javax.xml.bind.annotation.*;


Lets remove the * and add all the dependencies

This is an auto generated one

BLasan mentioned this pull request Sep 5, 2024

Add Custom Backend for APIs wso2/api-manager#3007

Closed

AnuGayan requested changes Sep 12, 2024

View reviewed changes

BLasan force-pushed the sequence-as-a-backend branch from 1fac89a to 3a6e34f Compare September 14, 2024 17:30

BLasan marked this pull request as ready for review September 19, 2024 04:52

BLasan force-pushed the sequence-as-a-backend branch from 3d896b0 to 4823c10 Compare September 19, 2024 05:51

BLasan mentioned this pull request Sep 21, 2024

[Doc]: Sequence as a Backend for REST APIs wso2/docs-apim#8386

Merged

BLasan mentioned this pull request Sep 23, 2024

Sequence backend 4.3.0 wso2/docs-apim#8389

Closed

AnuGayan requested changes Sep 23, 2024

View reviewed changes

BLasan force-pushed the sequence-as-a-backend branch from 36c3c1c to 00d6c1c Compare September 23, 2024 13:35

RakhithaRR reviewed Sep 26, 2024

View reviewed changes

AnuGayan reviewed Sep 26, 2024

View reviewed changes

BLasan force-pushed the sequence-as-a-backend branch from 0e4a9ac to 068529c Compare September 26, 2024 06:46

BLasan added 12 commits September 26, 2024 15:13

Init Commit

5c4b8b9

Add: New REST API to upload Custom Sequence

c5c9149

Add: Custom Sequence support for the APIS

b55dabb

Fix: Analytics Type conversion issue

74443fb

Revamp: Endpoint Config update for API import via APICTL and REST

65628a4

Fix: Sequence generating Issue

e7f8f2f

Changed: REST API of Custom Backend update

ab62a27

Add: Custom Backend to APICTL file created on revision deployment

64fb6e5

Remove: Unnecessary Codes

f16e756

Remove: Unwanted Codes in SynapseArtifactGenerator

011fdf0

Remove: APIDTO Sequence update

d3355fb

Remove: APIDTO Sequence update unwanted comments

081b0cf

BLasan added 25 commits September 26, 2024 15:26

Fix: Bugs in Revision scenarios

aa58c51

Add: API Validation for Sequence Backend

1798fa5

Add: NPE handling

63f10f6

Fix: Sequence Backend Download Issue

df82c20

Fix: Advertised only API Issue

63ae2cf

Add: Validations for APICTL Export scenario

2edd31e

Rename: Custom Backend dir to Sequence Backend

720909f

Rename: Custom Backend dir to Sequence Backend

ffac3a3

Add: Schema for all database types

a5c791b

Add: APICTL Project export Endpoint Configuration

3784395

Fix: Sequence Backend storing issue in APICTL project

87f2e12

Add: Composite Primary Key

1620f6c

Fix: JSON Parsing error with Number format exception

b758283

Fix: Null Pointer Exception for non endpoint configs

c47b485

Hadneld: Number Format Exceptions when parsing

052e53a

Revert: SESSION TIMEOUT CONFIG in Publisher CommonUtil

8bd45c3

Remove: Unused Import

58e7b20

Fix: Commented Issues

be1cb81

Fix: Input Stream Closing Issue

d1bb088

Fix: TemplateUtil Test error

15c6b2f

Handle: Transaction Rollbacks on failure

d7415a8

Fix: Type in debug log

67d5b98

Fix: Lint Issues

54ae42a

Add: Sequence Backend schema for MultiDC

43047d7

Fix: Merge Conflicts

d12fca0

BLasan force-pushed the sequence-as-a-backend branch from 1889651 to d12fca0 Compare September 26, 2024 10:08

AnuGayan approved these changes Sep 26, 2024

View reviewed changes

RakhithaRR merged commit c72f127 into wso2:master Sep 27, 2024
2 of 7 checks passed

This was referenced Oct 11, 2024

Sequence as backend wso2/docs-apim#8470

Merged

Add: Sequence Backend Test wso2/product-apim#13574

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add: Support Custom Sequence as a Backend of an API #12524

Add: Support Custom Sequence as a Backend of an API #12524

BLasan commented Aug 25, 2024 •

edited

Loading

CLAassistant commented Aug 25, 2024

AnuGayan Sep 12, 2024

AnuGayan Sep 12, 2024

AnuGayan Sep 12, 2024

AnuGayan Sep 12, 2024

AnuGayan Sep 12, 2024

AnuGayan Sep 12, 2024

BLasan commented Sep 19, 2024

BLasan commented Sep 21, 2024

BLasan commented Sep 23, 2024 •

edited

Loading

AnuGayan Sep 23, 2024

AnuGayan Sep 23, 2024

AnuGayan Sep 23, 2024

AnuGayan Sep 23, 2024

AnuGayan Sep 23, 2024

AnuGayan Sep 23, 2024

AnuGayan Sep 23, 2024

BLasan commented Sep 23, 2024

BLasan commented Sep 26, 2024

RakhithaRR Sep 26, 2024

RakhithaRR Sep 26, 2024

RakhithaRR Sep 26, 2024

AnuGayan Sep 26, 2024

AnuGayan Sep 26, 2024

BLasan Sep 26, 2024

		@@ -0,0 +1,142 @@
		package org.wso2.carbon.apimgt.rest.api.publisher.v1.dto;

Add: Support Custom Sequence as a Backend of an API #12524

Add: Support Custom Sequence as a Backend of an API #12524

Conversation

BLasan commented Aug 25, 2024 • edited Loading

Purpose

Approach

Affected Flows

Issue

CLAassistant commented Aug 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BLasan commented Sep 19, 2024

BLasan commented Sep 21, 2024

BLasan commented Sep 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BLasan commented Sep 23, 2024

BLasan commented Sep 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BLasan commented Aug 25, 2024 •

edited

Loading

BLasan commented Sep 23, 2024 •

edited

Loading